3d Imaging in the Studio (and Elsewhere...)

At 3DV Systems Ltd. we developed and built a true 3D video camera (Zcam), capable of producing RGB and D signals where D stands for distance or depth to each pixel. The new RGBD camera makes it possible to do away with color based background substitution known as chroma-key as well as creating a whole gallery of new effects and applications such as multilayer foreground as well as background substitutions and manipulations. The new multilayerd modality makes possible the production of mixed reality real time video as well as postproduction manipulation of recorded video. The new RGBD camera is scannerless and uses low power laser illumination to create the D channel. Prototypes have been in use for more than 2 years and, are capable of sub-centimeter depth resolution at any desired distance up to 10 m. on the present model. Additional potential applications as well as low cost versions are currently being explored. 1.INTRODUCTION The art of creating video films by combining images taken in the studio and images created by graphic artists on a computer or by hand or by an additional camera is an important part of video production. In many applications chroma-key background subtraction is used in the studio as a mean of replacing the background by any desired image both artificial and real. The method is extensively documented in the literature some examples are: [1], [2], [3]. The chroma-key technique has the advantages of being simple to comprehend and yield very good quality images that made it popular it can achieve image implantation to a precision level of a few pixels. Beside the advantages, the method has some drawbacks listed next that can be eliminated by using our new camera as will be shown. Single layer interchange Only layers in the back can be changed Requires blue background color (blue usually) screen Requires studio environment Actors cannot use blue dresses or blue objects If however, the video producer wishes to insert into the field of view layers and objects located at any desired distances from the camera the c-k method is incapable of doing it. This limitation is even further enhanced if one wishes to keep the existing background in addition to the implanted layers. 3DV has developed and patented a robust method for performing multilayerd implantation and replacement has to be based on complete data of the field of view (FOV) that is the RGB data as well as the D distance data for each pixel in the FOV. The new camera known under its trademark name as Zcam is hence a true 3D video camera. The concept behind the camera is very versatile and may be applied to many fields, some of which will be discussed later. The two figurers (1 & 2) present schematically a typical scenario of replacing and inserting layers according to their relative range position vis-à-vis the camera. Three-Dimensional Image Capture and Applications IV, Brian D. Corner, Joseph H. Nurre, Roy P. Pargas, Editors, Proceedings of SPIE Vol. 4298 (2001) © 2001 SPIE. · 0277-786X/01/$15.00 48 Fig. 1: The original scene to be modified Fig. 2: The scene after the modification Proc. SPIE Vol. 4298 49 2. THE CONCEPTS BEHIND THE CAMERA The basic tool used for the placement and replacement of layers in the FOV without resorting to c-k is a special camera that delivers for each pixel in the FOV in addition to RGB an additional parameter D indicating distance. In a sense this is a camera capable of generating 3 dimensional images of the FOV or to be more precise 21/2 dimensions when looking from a single vantage point. The unique camera is capable of doing so at video rate and is compatible with all existing standards and formats. The concept of operation is based on generating a "light wall" having a proper width moving along the FOV. The said “light wall “ can be generated for example as a square laser pulse of short duration having a field of illumination (FOI) equal to the FOV (fig. 3). As the light wall hits the objects in the fov it is reflected back towards the camera carrying an imprint of the objects. The imprint contains all the information required for the reconstruction of the depth map. Fig. 3: “light wall” moving along the FOV. Fig. 4: imprinted light wall returning to camera. Proc. SPIE Vol. 4298 50 The 3D information can now be extracted from the reflected deformed "wall" by deploying a fast image shutter in front of the ccd chip and blocking the incoming light as shown in fig. 5. Fig. 5: Truncated "light wall"front cut The collected light at each of the pixels is inversely proportional to depth of the specific pixel. As an alternative it is possible to retain the rear section of the reflected wall (fig. 6) and thus get the “negative” of the depth. Since reflecting objects in the FOV may have any reflectivity coefficient there exist a need to compensate this effect and hence a normalization procedure is introduced. Fig. 6 rear portion of the reflected light wall (rear cut). Proc. SPIE Vol. 4298 51 The normalized depth of pixel (i,j) can be calculated by simply dividing the front portion pixel intensity by the corresponding portion of the total intensity: R(i, j) = I (i,j) front / I (i,j) total Hence resulting reflectivity normalization procedure, which completes the capture of the depth, map that comes out as a conventional black/white image. Fig. 9, which comes next, is a very illustrative graphical representation of the kinematics of the light ray corresponding to a single pixel in the image much like the diagrams used for explaining the special theory of relativity.